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AN  EXTENSION  OF  KENMLVF  CONCORDANCE  TEST 
WHERE  TIES  ARE  ALLOWED 


\  INTRODUCTION 

Kendall's  measure  of  concordance  and  associated  probability  tables 
establishes  critical  regions  for  testing  the  null  hypotbesis  of  random 
rankings  of  M  items  by  N  judges.' "H"  The  M  items  are  ranked  according  to  an 
agreed  upon  criterion,  such  as  beauty  and  cost  effectiveness,  or  to  the 
efficacy  of  a  compression  scheme,  which  was  the  basis  for  this  Research 
Note.  The  concordance  probability  values  are  approximated  by  the  F- 
distribution  and  the  tests  require  that  each  judge  produce  a  valid  ranking 
of  the  M  item,  i.e.  ties  '’mong  the  items  are  not  allowed.  The  purpose  of 
this  work  was  to  develop  exacc  probability  tables  for  a  limited  number  of 
values  of  the  parameters  N  and  M  where,  in  fact,  ties  were  allowed. 

BACKGROUND  ■ 

2 

The  purpose  of  the  original  study  was  to  determine  which  of  several 
compression  techniques  was  best  in  the  sense  that  it  produced  the  most 
acceptable  digital  image  when  the  compressed  image  was  decoded  and 
displayed  on  a  TV  at  8  bits.  Five  compression  techniques  labeled  C2 
through  C6  were  evaluated  along  with  the  uncompressed  image  labeled  Cl. 
The  compression  techniques  as  well  as  the  Image  types  are  not  relevant  to 
this  study.  In  the  original  study,  the  null  hypothesis  of  no  difference 
in  the  effects  of  compression  was  tested  by  the  chi-square  test. 

Each  test  image  was  compressed,  then  decompressed,  and  finally  stored 
in  DIAL  as  an  8--bit  image.  Six  compressions  of  each  image  were  stored  in 
DIAL  for  subsequent  viewing  and  comparison.  There  were  45  such  images 
used  in  the  experiment.  Each  interpreter  (there  were  12)  was  required  to 
choose  either  the  right  or  the  left  image  when  a  pair  was  displayed  side 
by  side  on  a  TV  display.  Thus,  each  interpreter  made  =  15 

comparisons  of  the  six  processing  schemes  (five  compressions  and  one 
original)  for  each  of  the  45  images.  The  pairs  to  be  compared  a**  well  as 
the  image  type  were  presented  in  a  random  manner  to  the  interpreter.  The 


*M.  Hammerton,  Statistics  for  the  Human  Sciences,  London  and  New  York; 
Longman,  1975. 


^SPIRIT  II:  Special  Imagery  Recognition  and  Interpretation  Tests,  Final 
Technical  Report,  Prepared  for  U.S.  Army  Engineer  Topographic 
Laboratories,  Fort  Belvoir,  VA.  Prepared  by  Autometric,  Incorporated, 
Falls  Church,  VA ,  June  1982, 
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rank  score  for  each  processing  scheme  was  the  totality  of  the  choices  over 
image  and  over  interpreter.  Thus,  there  were  15*45*12  =  8100  choices 
made,  and  under  the  null  hypothesis,  the  expected  value  for  each 
processing  scheme  was  8100/6  =  1350.  The  chi-square  test  is  valid  for 
this  experiment  since  the  expected  value  of  each  of  the  six  cells  is  well 
above  5. 

Today,  because  of  relatively  low  cost  CPU  time,  there  is  little  reason 
why  experimental  results  should  be  tested  with  approximate  probability 
distributions. J  It  would  appear  that  although  the  chi-square  test  was  a 
valid  test,  it  provided  very  little  information,  especially  when  the  large 
number  of  tests  is  considered.  Kendall's  concordance  test,  which  is  more 
nearly  appropriate,  enables  the  null  hypothesis  of  random  ranking  of  the 
six  processing  schemes  by  the  12  interpreters  to  be  tested.  However, 
Kendall's  measure  of  concordance  cannot  be  used  since  the  theory  does  not 
allow  ties,  and  the  like'1  ihood  of  ties  is  high  in  the  experimental 
procedure  described  above  vsee  appendix  A).  The  purpose  of  this  study  is 
to  demonstrate  how  exact  probability  values  that  pertain  to  the  experiment 
at  hand  can  be  calculated  and  used.  It  should  be  noted  that  the  derived 
probability  tables  (sec  appendix  B)  are  general,  and  they  can  be  applied 
to  a  variety  of  similar  experiments. 

NUMERICAL  TESTS 

The  history  of  the  comparison  tests  was  reorganized  into  a  3- 
dimensional  array  and  stored  on  disc  for  subsequent  analysis.  The  first 
dimension  of  the  array  specified  the  L  =  1,  45  images;  the  second 
dimension  specified  the  K  =  1,12  interpreters;  and  the  third  dimension 
defined  the  I  =  1,36  values  associated  with  the  (L,K)^  image-interpreter 
event.  The  values  that  pertain  to  the  6  -  x  6-score  matrix  are  given  in 
table  10. 


O 

B.  Efron,  "Computers  and  the  Theory  of  Statistics.  Thinking  the 
Unthinkable,"  SIAM  Review,  Vol.  21,  No.  4,  October  3979. 
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TABLE  1.  SCORE  MATRIX 


1 

v12 

v13 

v14 

v15 

v16 

V21 

1 

v23 

v24 

v25 

v26 

V31 

v32 

1 

v34 

v35 

v36 

V41 

v42 

v43 

1 

v45 

v46 

V51 

v52 

v53 

v54 

1 

v56 

V61 

v62 

v63 

v64 

v65 

1 

If  Vjj  =  1,  then  the  compression  was  judged  to  be  superior  to  the 
compression.  Note  that  if  Vjj  =  1,  then  Vjj  =  0.  The  ones  along  the 
diagonals  are  added  to  each  score  so  that  scores  will  range  from  1  to  6. 
The  ^  score  is  determined  by  summing  the  values  in  the  I1"  row. 

Two  statistical  tests  were  conducted  using  the  image  comparison 
histories  des<  :ibed  above.  The  first  test  was  organized  to  determine 
whether  the  rankings  of  the  compression  schemes  were  random.  The  second 
experiment  was  developed  to  test  whether  there  was  a  difference  between 
the  rankings  as  determined  by  beginners  when  compared  to  the  rankings  as 
determined  by  experts.  Six  of  the  12  interpreters  were  regarded  as 
experts,  and  6  were  regarded  as  beginners,  or  non-experts.  The  non¬ 
experts  were  given  the  same  background  information  as  the  experts. 

Forty-five  independent  rank  tests  were  performed  by  12  image 
interpreters.  The  45  tests  pertain  to  the  45  images  that  were  subjected 
to  six  (one  baseline  and  five  compression  schemes)  digital  processing 
exercises.  When  all  six  compressions  were  evaluated,  the  average  ranking 

over  all  45  images  by  the  12  interpreters  turned  out  to  be  the  following: 

Cl  :  1.49 
C2  :  4.18 
C3  :  5.96 
C4  :  3.02 
C5  :  3.21 
C6  :  3.12 

The  smallest  J-statistic  was  1206  (image  (*21),  which  is  beyond  all  of  the 
entries  for  N  =  12  in  table  B8.  The  hypothesis  of  a  random  ranking  of  the 
six  compression  schemes  by  the  12  observers  is  totally  unacceptable. 

The  interpreters  were  extremely  consistent  in  ranking  C3  last,  and  to 
a  lesser  degree,  they  were  consistent  in  ranking  Cl  (baseline)  first,  and 

C2  fourth.  Note  that  C4,  C5,  and  C6  seem  to  be  tied.  The  comparison  data 


was  than  processed  to  determine  whether  the  hypothesis  of  random  ranking 
was  tenable  when  five  compression  schemes  were  evaluated.  This  was  done 
three  times,  where  in  turn.  Cl,  C2,  and  C3  were  eliminated  from 
consideration.  The  averaged  rankings  turned  out  to  be  the  following: 


Cl  out 

C2  out 

C3  out 

Cl 

1.40 

1.49 

C2 

3.27 

4.16 

C3 

4.96 

4.99 

C4 

2.16 

2.78 

3.01 

C5 

2.35 

2.96 

3.22 

C6 

2.25 

2.87 

3.12 

The  smallest  J-statistic  when  Cl  was  removed  was  5 64  (image  #11),  and  when 
C2  was  removed,  the  smallest  J-statistic  was  774  (image  #21).  Both  of 
these  values  are  beyond  all  of  the  entries  for  N  =  12  in  table  B7.  The 
smallest  J-statistic  when  C3  was  removed  was  126  (image  #21),  and  from 
table  B7  the  statistic  is  significant  at  the  0.926  probability  level. 
Although  this  calculation  does  not  demonstrate  randomness  in  the  rankings, 
it  does  indicate  a  trend  toward  randomness  when  the  highly  consistent 
ranking  of  C3  being  last  is  removed  from  consideration.  Several  other 
image  comparisons  provided  J-statistics  that  were  not  beyond  the  tabulated 
entries.  For  example,  the  next  smallest  J-statistic  was  238  (images  #33 
and  #36),  where  from  table  B7  the  significance  level  is  0.998. 

The  next  step  was  to  remove  the  three  possible  pairs  of  Cl,  C2,  and  C3 
to  determine  whether  the  hypothesis  of  random  ranking  was  tenable  when 
four  compression  schemes  were  evaluated.  The  averaged  rankings  turned  out 
to  be  the  following: 


Cl  and  C2  out 

Cl  and  C3  out 

C2  and  C3  out 

Cl 

C2 

C3 

3.99 

3.25 

1.40 

C4 

1.92 

2.15 

2.78 

C5 

2.09 

2.35 

2.95 

C6 

2.00 

2.25 

2.87 

The  smallest  J-statistic  when  Cl  and  C2  were  removed  was  350  (image 
#13).  This  value  is  beyond  all  of  the  entries  for  N  =  12  in  table  B6. 
The  smallest  J-statistic  when  Cl  and  C3  were  removed  was  6,  which  is  at 
the  0.107  significance  level.  In  fact,  20  of  the  J-statistics  were  at  or 
below  SO,  which  is  at  the  0.950  significance  level.  The  smallest  J- 
statistic  when  C2  and  C3  were  removed  was  54,  which  is  at  the  0.804 
significance  level.  All  other  J-statistics  were  well  beyond  the  0.950 
significance  level.  The  hypothesis  of  a  random  element  or  a  lack  of 
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concordance  among  the  12  interpreters  was  evident  when  the  consistently 
bad  compression  C3  and  the  consistently  good  compression  Cl  were  removed 
from  consideration. 


The  next  step  was  to  remove  Cl ,  C2 ,  and  C3  to  determine  whether  the 
hypothesis  of  random  ranking  was  tenable  when  the  three  compression 
schemes  C4,  C5,  and  C6  were  evaluated*  The  averaged  ranking  over  the  12 
interpreters  and  45  images  turned  out  to  be  the  following: 


C4  : 

1.91 

C5  : 

2.08 

C6  : 

2.00 

The  largest  J-statistic  was  56,  which  from  table  B2  is  at  the  0.967 
significance  level.  All  other  J-statistics  are  less  than  or  equal  to  54, 
which  is  at  the  0.950  significance  level.  The  hypothesis  of  a  lack  of 
concordance  is  definitely  in  evidence  here.  Finally,  C4,  C5,  and  C6  were 
removed  from  consideration  and  the  averaged  rankings  over  the  12 
interpreters  and  45  images  turned  out  to  be  the  following: 

Cl  :  1.09 
C2  :  1.94 
C3  :  2.97 

The  smallest  J-statistic  was  206,  which  is  beyond  all  of  the  entries  for  N 
=  12  in  table  B2.  The  hypothesis  of  a  random  ranking  must  be  rejected  in 
this  case. 


The  second  test  utilized  existing  theory  to  test  whether  the  six 
experts  were  in  agreement  with  the  six  non-experts.  Hotelling's  T^ 
statistic  was  used  to  test  the  equality  of  averaged  rankings  between  the 
two  groups.  The  abandonment  of  the  mathematical  purity  proclaimed  in  the 
Background  Section  is  in  this  case  only  a  minor  accommodation  to 
expediency.  Except  for  the  assumptions  of  multivariate  normal 
distributions  and  equal  covariance  matrices,  the  conditions  associated 
with  the  experiment  are  identical  to  the  requirements  of  the  two  sample  T^ 
statistics.  The  tests  on  the  equality  of  the  average  rankings  between 
experts  and  non-experts  were  performed  over  the  same  experimental 
conditions  described  above  in  the  ranking  tests.  When  the  null  hypothesis 
of  equal  means  is  true,  then  the  quantity  F  given  below  has  the  F- 
distribution  with  stated  degrees  of  freedom. 


^D.  Morrison,  Multivariate  Statistical  Methods,  McGraw  Hill  Book  Company, 
New  York,  1967. 
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F  =  1I-M  t2 
10 

with  aegrees  of  freedom  M  and  11-M 
T2  =  3DTS_1D 

D  =  Mean  Vector  Difference 
S  Pooled  Covariance  Matrix 

When  the  mean  differences  associated  with  all  six  compression  schemes 
were  tested,  none  of  the  45  test  statistics  exceeded  the  Fg  ^  -  3.11,  90 
percent  significance  level.  In  the  case  where  Cl  was  ’removed  from 
consideration,  none  of  the  45  test  statistics  exceeded  the  F^  g  =  3.40,  90 
percent  significance  level.  When  C2  was  removed,  the  test  statistic 

associated  with  image  #42  was  significant,  and  whin  C3  was  removed,  the 
test  statistic  associated  with  images  #36  and  #42  was  significant.  In  the 
case  where  Cl  and  C2  were  removed  from  consideration,  the  test  statistic 
associated  with  image  #27  exceeded  the  y  =  3.98,  90  percent 
significance  level.  When  Cl  and  C2  were  removed,  the  test  statistic 

associated  with  image  #36  was  excessive,  and  when  C2  and  C3  were  removed, 

the  test  statistic  associated  with  image  #42  exceeded  the  95  percent  level 

of  significance  (F^  y  =6.09).  When  Cl,  C2,  and  C3  were  removed  from 
consideration,  none  of  the  45  test  statistics  exceeded  the  Fg  g  =  5.25,  90 
percent  significance  level. 
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DISCUSSION 


The  mathematician  will  recognize  that  much  of  the  work  here  is  a 
fishing  expedition  rather  than  executing  a  previously  designed  statistical 
experiment.  It  must  be  recalled,  however-,  that  the  main  purpose  of  the 
work  was  to  demonstrate  that  the  analyst  need  not,  in  !:his  day  of 
relatively  inexpensive  computer  CPU  time,  resort  to  a  Procrustean  method 
when  analyzing  experimental  results.  In  fact,  exact  cumulative 
probability  tables  were  developed  for  the  experiment  at  hand.  That  a 
great  deal  of  searching  through  the  data  and,  in  a  few  cases,  resorting  to 
Procrustean  methods  to  obtain  certain  results  is  readily  admitted. 

The  various  tests  for  random  ranking  among  the  12  interpreters  showed 
that  a  strong  consistency  among  the  interpreters  for  several  of  the 
compressions  masked  a  randomness  or  lack  of  concordance  among  the 
remaining  compressions.  When  all  six  compression  schemes  were  considered, 
the  null  hypothesis  of  random  ranking  among  the  12  interpreters  was 
decidely  rejected.  Whereas,  when  Cl,  C2,  and  C3  were  removed  from 
consideration,  only  1  of  the  45  J-statistics  exceeded  the  95  percent 
significance  level.  This  result  cannot  be  used  to  reject  the 
hypothesis.  Note  that  the  probability  of  getting  at  least  1  value  out  of 
45  beyond  the  95  percent  significance  point  is  0.900  when  the  null 
hypothesis  is  true.  This  result  is  derived  from  the  cumulative  binomial 
distribution  since  under  the  null  hypothesis  the  J-statistic  values  are 
independent  and  there  is  a  five  percent  chance  for  each  to  fall  in  the 
critical  region.  The  tests  for  random  ranking  among  the  12  interpreters 
also  showed  that  a  definite  lack  of  concordance  among  the  interpreters  for 
several  of  the  compressions  tended  to  mask  a  strong  consistency  among  the 
remaining  compressions.  This  was  shown  when  C4,  C5,  and  C6,  were  removed 
from  consideration. 

The  several  tests  to  determine  if  there  were  any  differences  between 
expert  interpreters  and  non-expert  interpreters  did  not  reject  the 
hypothesis  of  no  difference.  When  C2  was  removed  from  consideration,  two 
sample  F  statistics  exceeded  the  90  percent  significance  level.  Note  that 
in  this  case  the  probability  of  getting  at  least  2  values  out  of  45  beyond 
the  90  percent  significance  level  is  0.948  when  the  null  hypothesis  of 
equal  means  is  true.  When  Cl,  C2,  and  C3  were  removed  from  consideration, 
none  of  the  45  values  were  beyond  the  90  percent  critical  point.  Under 
the  null  hypothesis  of  equal  means,  the  likelihood  of  this  result  is  only 
0.009.  Either  an  unlikely  occurrence  has  taken  place  or  the  assumption  of 
a  multivariate  normal  distribution  is  exaggerated. 

The  inconsistency  noted  in  the  last  paragraph  is  another  reason  for 
using  exact  probability  distributions  wherever  possible.  In  order  for  us 
to  get  an  idea  about  the  random  (or  non-random)  nature  of  the  C4,  C5,  and 
C6  ranking  results,  the  12  *  45  =  540  3-component  ranks  were  extracted 
from  the  comparison  history  and  summarized  in  tables  2  and  3. 
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TABLE  2.  EXPERT  RESULTS 


RANKINGS 


Interpreter 

(222) 

(123) 

(132) 

(213) 

(231) 

(312) 

(321) 

1 

10 

7 

11 

4 

4 

5 

4 

2 

15 

2 

5 

5 

12 

2 

4 

3 

13 

6 

6 

7 

4 

4 

5 

4 

9 

8 

8 

5 

7 

6 

2 

5 

7 

10 

3 

7 

6 

6 

6 

6 

10 

6 

5 

8 

6 

3 

7 

Totals 

64 

39 

38 

36 

39 

26 

28 

Prob.  Est. 

0.237 

0.144 

0.141 

0.133 

0.144 

0.G96 

0.104 

TABLE  3.  ROM  ESPERT-RESULfS 


RANKINGS 


Interpreter 

(222) 

(123) 

(132) 

(213) 

(231) 

(312) 

(321) 

1 

9 

6 

6 

6 

7 

8 

3 

2 

i2 

4 

5 

10 

8 

2 

4 

3 

11 

4 

1.0 

4 

10 

4 

2 

4 

12 

7 

5 

7 

6 

C 

3 

5 

9 

4 

8 

3 

8 

6 

7 

6 

11 

2 

15 

4 

4 

3 

6 

Totals 

64 

'  27 

49 

34 

43 

28 

25 

Prob.  Est. 

0.237 

0.100 

0.181 

0.126 

0.159 

0.104 

0.095 

If  the 

rankings  were  entirely  random,  then 

from 

appendix 

A  the 

inconsistent 

rankings 

(2,2,2 

)  should 

occur  with 

a  probability  of 

0.250; 

whereas. 

the 

six  consistent 

rankings 

should  each 

occur  wich  a  probability 

of  0.125. 

If 

random 

ranking 

is  assumed,  as  described  in 

appendix 

A,  then 

the  reserved  ecunts  of  the  several  rankings  when  compared  to  the  expected 
count  can  be  tested  by  using  the  chi-square  goodness  of  fit  test.  The 
expert's  X  ^  estimate  was  5.3,  which  is  not  significant;  whereas,  the  non¬ 
expert's  value  is  15.1,  which  is  significant  at  the  99  percent  level.  The 
expert's  results  demonstrate  more  consistency  than  the  non-expert's 
results.  both  sets  of  results  appear  to  show  a  reluctance  in  allowing 
compression  C4  to  be  ranked  third. 


It  can  be  concluded  that  C4,  C5,  and  CA  are,  for  all  practical 
purposes,  equivalent  under  the  experimental  conditions  and  that  Cl  >  (C4, 
C5,  and  C6)  >  C2  >  C3.  The  strong  relationship  (Cl  >  C2  >  C3)  tended  to 
deny  a  lack  of  concordance  when  the  original  null  hypothesis  of  the  six 
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schemes  being  random  was  tested.  The  sub-hypothesis  of  C4,  C5,  and  C6 
producing  a  lack  of  concordance  was  supported  by  the  data  when  Cl,  C2,  and 
C3  were  excluded  from  the  analysis.  Other,  unrelated,  multivariate 
analysis’  work  at  ETL  has  also  produced  uncertain  or  ambiguous  results. 
In  the  referenced  work,  it  was  determined  that  a  simple  2-component 
signature  did  just  as  well,  and  in  some  cases  better,  than  a  large 
component  descriptor  in  segmenting  an  aerial  image.  There  is  a  suspicion 
here  that  the  linear  models  used  in  multivariate  analysis  do  not 
adequately  represent  the  unknown  structure  relating  variables  of  interest. 

CONCLUSIONS 

1.  Experimental  tests  should  use  an  exact  statistical  design  and  develop 
relevant  probability  data  when  needed. 

2.  An  extension  of  Kendall's  probability  of  concordance  was  developed  for 
a  limited  number  of  parameters  and  shown  to  he  useful. 

3.  There  is  a  growing  concern  over  the  validity  of  many  multivariate 
analyses  using  large  numbers  of  components,  where  unknown  internal 
relationships  tend  to  contaminate  results. 


■*M.  Crombie,  N.  Friend,  and  R.  Rand,  Feature  Component  Reduction  Through 
Divergence  Analysis,  U.S.  Array  Engineer  Topographic  Laboratories,  Fort 
Belvoir,  VA,  ETL-0305,  October  1982. 
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APPENDIX  A.  RANDOM  RANKING  OF  M  ITEMS  BY  N  'JUDGES 


The  numerical  ranking  of  M  items  is  determined  by  making  choices  over 
the  2  possible  binary  comparisons  of  the  M  items.  The  rank  of  a 
specific  item  is  determined  by  adding  1  to  the  numoer  of  times  the  item 
was  chosen.  Since  a  specific  item  compared  (M-])  times,  the  rank 
values  will  range  from  1  to  M.  Note  tnat  this  type  of  ranking  procedure 
can  produce  ties.  The  purpose  here  is  to  describe  the  development  of 
statistical  tables  for  testing  the  null  hypothesis  of  random  rankings 
among  N  judges. 


The  density  function  and  associated  distribution  function  were 
developed  using  Monte  Carlo  methods.  The  two  functins,  p(J)  and  P(J) 
respectively,  are  characterized  by  two  parameters,  namely  N,  the  number  of 
judges  and  M,  the  number  of  items  to  be  ranked.  If  the  experimental 
results  are  organized  into  N  rows  and  M  columns,  and  if  R^  is  the  expected 
sum  of  the  column,  then  the  random  variable  J  is  defined  as  follows: 


j  J,<RfV2 

1=1 


yth 


R' j  :  observed  sum  of  the  J  column 


Assuming  that  the  rankings  are  random,  then  each  of  the  N  judges' 
rankings  will  sum  to  M(M+l)/2  for  a  total  of  NM  (MH)/2.  Since  no  one  of 
the  items  is  favored,  the  expected  value  for  each  of  the  M  columns  is  R^  = 
N(M4-l)/2. 


For  example,  let  M  =  3  and  consider  the  following  matrix  that 

represents  results  from  the  I*"  judge: 


1 

v12 

v13 

VI  - 

V21 

1 

v23 

V31 

v32 

1 

If  v^g  =  1,  then  item  L  was  judged  to  be  superior  to  item  K.  Note  that  if 
VLK  =  ttien  vkl  =  0*  The  ones  along  the  diagonal  pertain  to  the  ones 
added  to  each  score.  The  rank  is  determined  by  summing  tht  J5"  row  of 
V.  There  are  2J  ways  that  ones  can  be  distributed  over  V.  In  general, 
there  are  2^M  ways  where  QM  =  M(M-l)/2.  There  are  3!  ways  that  consistent 
rankings  may  occur,  and  in  general,  there  are  M!  ways  that  consistent 
rankings  may  occur.  The  eight  possible  VT  are  listed  as  follows: 


1113 
0  1  1  =  >2 

0  0  1  1 

10  0  1 

1  1  1  =  >3 

10  1  2 

110  2 
0  1  1  =  >2 

10  12 


10  0  1 

1  1  0  =  >2 

111  3 

110  2 

0  1  0  =  >1 

1113 

10  1  2 

1  1  0  =  >2 

011  2 


1113 
0  1  0  =  >1 

0  112 

10  1  2 
1  1  1  =  >3 

0  0  1  1 


Note  that  the  latter  two  arrays  are  transposes  of  one  another  and  in  fact 
are  logically  inconsistent.  The  first  array  implies  that  the  first  item 
was  judged  superior  to  the  second  item,  which  in  turn  was  judged  superior 
to  the  third  item.  The  array  also  implies  that  the  third  item  was  judged 
superior  to  the  first  item. 


The  distribution  of  J  under  the  null  hypothesis  was  developed  by 
generating  pseudo  random  numbers  from  one  to  eight  and  then  selecting  one 
of  the  eight  possible  rankings.  This  exercise  was  performed  N  times  to 
produce  one  value  of  J.  A  large  number  of  J-values  were  generated  in  this 
manner  to  estimate  the  distribution  function. 

When  there  are  M  =  4  items,  there  are  2^  =  64  possible  rankings  of 
which  4 !  =  24  are  consistent.  The  24  consistent  rankings  and  40  possible 
ties  were  organized  into  a  (64  X  4)  array  and  sampled  by  generating  a 
random  number  between  1  and  64.  As  before,  the  exercise  was  performed  N 
times  to  produce  one  value  of  J. 


Two  sets  of  distribution  values  were  generated  for  M  =  3  and  for  M  = 
4.  In  the  first  case,  ties  were  not  allowed,  and  in  the  second  case,  they 
were  allowed.  When  there  are  M  =  5  items,  there  are  2*"  =  1024  possible 
rankings,  of  which  5!  =  120  are  consistent.  The  simple  enumeration  of  all 
possible  cases  was  discarded  in  favor  of  the  sampling  procedure  described 
next.  If  M  =  5,  then  M(M-l)/2  =  10  random  values,  either  one  or  zero, 
were  developed  and  inserted  into  Vj  according  to  the  rules  defined  above 
for  M  =  3.  This  procedure  was  repeated  N  times  to  produce  one  value  of 
J.  This  procedure  was  used  for  M  =  5  and  for  M  =  6.  In  these  cases, 
distribution  values,  where  ties  were  allowed,  were  calculated,  but  not 
values  where  ties  were  not  allowed. 


It  should  be  noted  that  rankings  are  equivalent  to  scores  in  this 
exercise.  Thus,  a  higher  score  implies  a  higher  rank.  If  an  ordinal 
ranking  is  desired,  then  the  scores  should  be  modified  by  the  relation  (M- 
S+l).  For  example,  if  M  =  5  and  a  particular  scoring  was  (1,3, 5,4, 2), 
then  the  equivalent  ordinal  ranking  is  (5, 3, 1,2, 4). 
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APPENDIX  B.  PROBABILITY  TABLES 


Monte  Carlo  methods  were  used  to  generate  the  cumulative  probability 
values  given  in  tables  B1  through  B8.  Tables  B1  and  B2  pertain  to  N  -  1 
to  12  judges  and  M  =  3  items.  The  eight  possible  rankings  were  organized 
and  stored  in  the  same  order  as  described  in  appendix  A.  The  results  in 
table  Bl,  where  ties  were  not  allowed,  were  developed  by  random  sampling 
among  the  first  six  scores;  whereas,  all  eight  scores  were  sampled  for 
table  B2.  The  same  general  procedure  was  used  to  generate  tables  B3 
through  B6.  The  64  possible  rankings  were  organized  so  that  the  first  24 
scores  were  valid  scores,  and  the  last  40  scores  were  the  possible  ties. 

When  M  =  5,  there  are  120  consistent  scores  and  904  possible  ties,  and 
when  M  =  6,  there  are  720  consistent  scores  and  32,048  possible  ties.  The 
method  described  above  was  discarded  in  favor  of  the  second  method 
described  in  appendix  A  for  these  cases.  In  both  cases,  ties  were 
allowed;  however,  only  even  values  of  N  (up  to  N  =  12)  were  calculated  for 
M  =  6. 


In  all  cases,  400,000  J-values,  as  described  in  appendix  A,  were 
computed  to  develop  the  sample  density  functions  and  finally  the  sample 
cumulative  functions.  The  results  are  presented  to  three  decimal 
digits.  Several  of  the  probability  tables  were  generated  using  100,000  J- 
values,  and  the  vast  majority  of  these  values  differed  by  no  more  than  one 
in  the  third  decimal  place  from  those  generated  from  400,000  J-values. 
The  largest  discrepancy  was  three  digits  in  the  third  decimal  place. 


H  =  3  : 

TIES  .NOT  ALLOWED 

6 

N 

2 

8 

9 

10 

U 

il 

— 

.044 

.036 

-032 

.029 

.026 

.024 

.022 

.261 

.231 

.206 

.187 

.170 

.156 

.145 

.430 

.381 

.347 

.314 

.290 

.268 

.249 

.570 

.514 

.470 

.431 

.399 

.371 

.346 

.  748 

.696 

.646 

.603 

.564 

.531 

.499 

Tabla  B  2 


TIES  ALLOWED 


,  9  .  l  ,  ii. 


M 


3  : 


0 

2 

6 

8 

14 

18 

24 

20 

32 

3S 

42 

50 

54 

56 

62 

72 

74 

78 

86 

96 

98 

104 

114 

122 

126 


3 

4 

5 

6 

7 

8 

9 

JO 

JJ_ 

J2 

109 

.085 

.068 

.058 

.050 

.044 

.039 

.036 

.032 

.030 

566 

.  465 

.395 

.  344 

.302 

.270 

.244 

.224 

.206 

.  191 

777 

.675 

.595 

.534 

.480 

.437 

.400 

.370 

.345 

.321 

918 

.830 

.753 

.687 

.628 

.  579 

.536 

.500 

.  467 

.438 

988 

.948 

.900 

.850 

.801 

.755 

.713 

.674 

.640 

.607 

000 

.978 

.943 

.903 

.861 

.819 

.779 

.743 

.709 

.476 

.  987 

.962 

.929 

.865 

.853 

.822 

.789 

.756 

.725 

.998 

.939 

.972 

.950 

.924 

.897 

.869 

S42 

.814 

1.000 

.995 

.933 

.965 

.944 

.921 

.897 

.872 

.  843 

.998 

.992 

-982 

.968 

.951 

.933 

.914 

.891 

1.000 

.998 

.993 

.984 

-.973 

..960 

.945 

.929 

.999 

.995 

.988 

.979 

.968 

.955 

.940 

.999 

.996 

.99! 

.984 

.974 

.963 

.950 

1.000 

.999 

.995 

.991 

.985 

.976 

.967 

1.000 

.998 

.995 

.991 

.985 

.978 

.999 

.996 

-993 

.988 

.982 

.999 

..998 

.995 

.992 

.988 

1.000 

.999 

.997 

.995 

.992 

1.000 

.999 

.997 

.995 

.999 

.998 

.996 

1.000 

.999 

.998 

.999 

.999 

1.000 

.999 

.999 

1.000 
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Table  B  3 

K  =  4; N  ODD. 

TIK.>  HOT  ALLOWED 

V! 

J 

3 

5 

/ 

9 

11 

T 

.042 

.026 

.oTj 

.012 

.009 

3 

.030 

.056 

.037 

.027 

.020 

3 

.271 

.  143 

.093 

.068 

.052 

9 

.391 

.229 

.155 

.114 

.090 

i! 

.475 

.292 

.200 

.  i49 

.11/ 

i3 

.554 

.348 

.242 

.18! 

.143 

i7 

.659 

.439 

.315 

.239 

.  190 

19 

.  /OO 

480 

.349 

.267 

.213 

21 

.79' 

.556 

.411 

.318 

.256 

25 

.825 

.593 

.443 

,3i6 

.280 

27 

.853 

.629 

.477 

.375 

.305 

29 

.925 

.702 

.544 

.435 

.358 

33 

.946 

.740 

,583 

.470 

.389 

35 

.967 

.774 

.618 

.503 

.419 

37 

.982 

.790 

.635 

.519 

.433 

41 

.998 

.838 

.691 

.574 

.484 

43 

.999 

.848 

.704 

.537 

.496 

45 

1.000 

.877 

.739 

.624 

.531 

49 

.893 

.761 

.647 

.554 

51 

.906 

.780 

.668 

.574 

53 

.924 

.806 

.695 

.602 

57 

.933 

.820 

.712 

.618 

59 

.945 

.840 

.735 

.642 

61 

.955 

.858 

.756 

.665 

65 

.966 

.878 

.782 

.693 

67 

.968 

.883 

.787 

.699 

69 

.977 

.900 

.809 

,/24 

73 

.930 

.907 

.819 

.735 

75 

.983 

915 

.830 

.747 

77 

.988 

.927 

.847 

.766 

3i 

.991 

.937 

.863 

.785 

83 

•  9°3 

944 

.873 

.798 

35 

.99' 

.948 

.880 

.806 

89 

..-97 

.959 

.897 

.828 

91 

.998 

.96? 

.902 

.835 

93 

.998 

.965 

.907 

.841 

97 

.993 

.967 

.912 

.347 

99 

.999 

.570 

.918 

.855 

101 

.999 

.977 

.931 

.873 

105 

1.000 

.980 

.938 

.882 

107 

.982 

.942 

.889 

109 

.984 

.946 

.895 

113 

.987 

.  351 

.902 

115 

.988 

.954 

.906 
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Table  3  3  (Continued;  M  =*  4,  N  ODD:  TIES  NOT  ALLOWED 


J 

117 

121 

123 

125 

129 

131 

133 

137 

139 

141 

145 

147 

149 

153 

155 

157 

161 

163 

165 

169 

171 

173 

177 

179 

131 

185 

187 

:«4 

193 

195 

197 

201 

203 

205 

209 

211 

213 

217 

219 

221 

225 

227 

229 

233 


7 

9 

11 

.990 

.959 

.914 

.991 

.962 

.919 

.992 

.964 

.922 

993 

.968 

.930 

.995 

.973 

.937 

996 

.976 

.943 

996 

.977 

.945 

997 

.979 

.949 

997 

.981 

.95! 

998 

.982 

.955 

998 

.984 

.958 

998 

.985 

.959 

999 

.987 

.964 

999 

.989 

.967 

999 

.990 

.970 

999 

.990 

.971 

999 

.992 

.975 

999 

.992 

.975 

000 

.993 

.977 

.993 

.978 

.994 

.980 

.995 

.932 

.■>95 

.983 

.996 

.984 

.990 

.985 

.997 

.987 

.99' 

.987 

.998 

.989 

.998 

.989 

.998 

.990 

.998 

.991 

.998 

.  992 

.999 

.992 

.999 

.953 

.999 

.994 

.  °99 

.994 

■<nn 

.995 

.999 

.995 

.999 

.995 

.999 

.996 

1.000 

.996 

.997 

.997 

.997 
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Table  3  3  (Continued) 


H  =  4,  N  ODD:  TIES  SOT  ALLOWED 
N 


J 

235 

3 

5 

l 

9 

11 

237 

.997 

241 

.998 

243 

.998 

245 

.998 

249 

.998 

251 

.998 

253 

.998 

257 

.999 

259 

.999 

26! 

.999 

265 

.999 

267 

.999 

269 

.999 

273 

.999 

275 

.999 

277 

.999 

281 

.999 

1.000 
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Table  B  4  (Continued) 


M  =  4,  N  EVEN:  TIES  NOT  ALLOWED 


J 

294 

4 

6 

8 

10 

12 

296 

.999 

298 

.999 

300 

.999 

302 

.499 

304 

.999 

306 

.999 

308 

.999 

310 

.999 

312 

.99* 

314 

.999 

1.000 

23 


Tabl'*.  B  5 


X  =  4.  N  COD:  TIES  ALLOWED 


J 

3 

5 

7 

9 

11 

- 

.112 

058 

.036 

.026 

.019 

3 

.224 

.122 

.079 

.057 

.043 

5 

.475 

.282 

.190 

.  141 

.109 

9 

.64  7 

.420 

.296 

.224 

.177 

1! 

.747 

.512 

.372 

.285 

.227 

’3 

.821 

.587 

.436 

.339 

.273 

17 

.397 

.691 

•  >36 

.426 

.350 

19 

.924 

.733 

.579 

.466 

.384 

21 

.962 

.303 

•  65S 

.536 

.449 

25 

.973 

.833 

.690 

.571 

.483 

27 

.981 

.858 

.723 

.605 

.515 

29 

.993 

.906 

.788 

.674 

..582 

33 

.996 

.927 

.820 

.711 

.620 

35 

.998 

.944 

.848 

.744 

.655 

37 

.999 

.951 

.860 

.755 

.  670 

M 

1.000 

.969 

.896 

.807 

.723 

43 

.973 

.904 

.817 

.  735 

45 

.981 

.924 

•  84o 

.768 

49 

.  98o 

.936 

.863 

.789 

51 

.989 

.945 

.877 

.806 

53 

.992 

■956 

.896 

.829 

57 

.904 

-  Sf  2 

.906 

.842 

59 

.“96 

.969 

.919 

.859 

61 

.997 

.975 

.930 

.875 

o5 

.998 

.981 

.943 

.893 

67 

.998 

-982 

.945 

.897 

69 

.999 

-987 

.956 

.512 

73 

.999 

.989 

.959 

.913 

75 

.999 

.990 

.964 

•  92f 

77 

1.000 

.952 

.970 

.535 

81 

.994 

.975 

.944 

83 

.995 

.  979 

.950 

85 

.995 

.980 

.953 

*->9 

-99, 

.985 

.902 

91 

.998 

.987 

.965 

93 

.  99P 

.988 

.967 

97 

.998 

.989 

.969 

99 

.998 

.990 

.972 

101 

.999 

.993 

.978 

105 

.900 

.994 

.981 

107 

."99 

.995 

.982 

109 

l.uOO 

.99,6 

.984 

113 

.996 

.986 

1 1 5 

1  l  7 

.99 

.937 

4  1  7 

.997 

.989 
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Table  B  5  (Continued)  M  -  4.  N  ODD:  TIES  ALLOWED 


J 

121 

123 

125 

129 

131 

133 

137 

139 

141 

145 

147 

149 

153 

155 

157 

161 

163 

165 

169 

171 

173 

177 

179 

181 


9 

.997 

.998 

.998 

.999 

.999 

.999 

.999 

.999 

.999 

.999 

1.000 


jU 

.990 

.991 

.992 

.993 

.994 

.995 

.995 

.996 

.996 

.997 

.997 

.998 

.998 

.998 

.998 

.999 

.999 

.999 

.999 

-.999 

.999 

.999 

.999 

1.000 
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TIES  ALLOIfED 


10 


12 


Table  B  6  (Continued)  M  =  4,  H  EVEN:  TIES  ALLOWED 


N 


J 

4 

6 

8 

10 

12 

94 

1.000 

.994 

.980 

.956 

96 

.995 

.981 

.957 

98 

.996 

.984 

.962 

100 

.996 

.935 

.964 

102 

.996 

.986 

.966 

104 

.997 

.987 

.969 

106 

.997 

.989 

.971 

10S 

.998 

.989 

.973 

110 

.998 

.991 

.977 

114 

.999 

.993 

.979 

116 

.999 

.993 

.981 

113 

.999 

.994 

.983 

120 

.999 

.995 

.984 

122 

.999 

.996 

.986 

126 

1.000 

.996 

.988 

128 

.996 

.989 

130 

.997 

.989 

132 

.997 

.990 

134 

.998 

.992 

136 

.998 

.992 

138 

.998 

.993 

140 

.998 

.993 

142 

.°98 

.994 

144 

-S98 

.994 

146 

.999 

.995 

148 

.999 

.995 

150 

.999 

.996 

152 

.999 

.996 

154 

.999 

.996 

153 

.999 

.997 

160 

.999 

.997 

162 

.999 

.997 

164 

1.000 

.998 

166 

.998 

168 

.998 

170 

.998 

172 

.998 

174 

.998 

176 

.998 

178 

.999 

130 

.999 

182 

.999 

184 

.999 

186 

.999 

190 

.999 

192 

.999 
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Table  B  6  (Continued)  ilM.  N  EVEN 


TIES  ALLOWED 


Table 

B  7 

M  =  5: 

TIES 

ALLOWED 

J 

3 

4 

5 

6 

N 

7 

8 

9 

10 

11 

12 

0 

.004 

.002 

.001 

.001 

.001 

.001 

.000 

.000 

.000 

.000 

2 

.059 

.036 

.025 

.018 

.013 

.010 

.003 

.007 

.006 

.005 

4 

.  124 

.079 

.055 

.040 

.031 

.024 

.019 

.016 

.013 

,011 

6 

.228 

.151 

.  106 

.078 

.061 

.043 

.039 

.032 

.027 

.023 

8 

.310 

.210 

.  150 

.113 

.088 

.071 

.057 

.047 

.040 

.034 

10 

.433 

.308 

.227 

.173 

.137 

.110 

.09! 

.076 

.065 

.056 

12 

.471 

.336 

.249 

.191 

.15! 

.122 

.101 

.034 

.072 

.063 

14 

.537 

.436 

,333 

.261 

.210 

.172 

.  144 

.  121 

.104 

.091 

16 

.664 

.505 

.394 

.312 

.254 

.210 

.176 

.  149 

.  129 

.113 

18 

.713 

.559 

.442 

.354 

.291 

.242 

.204 

.173 

.151 

.132 

20 

.757 

.600 

.481 

.388 

,321 

.269 

.228 

.  194 

.170 

.149 

22 

.312 

.663 

.542 

.444 

.372 

.314 

.268 

.230 

.201 

.177 

24 

.844 

.701 

.581 

.481 

.406 

.345 

.296 

.255 

.224 

.  197 

26 

.392 

.765 

.648 

.547 

.466 

.400 

.346 

.300 

.265 

.234 

28 

.904 

.782 

.  667 

.  566 

.484 

.416 

.362 

.314 

.278 

.246 

30 

.925 

.814 

.702 

.602 

.519 

.450 

.393 

.342 

.304 

.271 

32 

.938 

.836 

.723 

.629 

.545 

.475 

.416 

.365 

.325 

.289 

34 

.959 

.874 

.776 

.681 

.597 

.526 

.464 

.409 

.367 

.323 

36 

.965 

.887 

.793 

.701 

.617 

.546 

.483 

.427 

.384 

.344 

38 

.975 

.908 

.822 

.733 

.651 

.580 

.516 

.459 

.414 

.372 

40 

.982 

.924 

.844 

.759 

.678 

.608 

.544 

.486 

.439 

,396 

42 

.935 

.932 

.857 

.774 

.695 

.625 

.561 

.502 

.455 

.411 

44 

.988 

.942 

.873 

.795 

.717 

.648 

.584 

.525 

.477 

.432 

46 

.993 

.958 

;.898 

.827 

.754 

.686 

.623 

.563 

.514 

.468 

48 

.994 

.961 

.905 

.836 

.763 

.696 

.633 

.574 

.525 

.478 

50 

.996 

.970 

.922 

.858 

.791 

,725 

,o64 

.605 

.555 

.508 

52 

.997 

.973 

.927 

.866 

.800 

.736 

.675 

.616 

.566 

.518 

54 

.998 

.979 

.939 

.883 

.821 

.759 

.699 

.642 

.592 

.544 

56 

.999 

.983 

.948 

.89? 

.837 

,778 

.721 

.663 

.613 

.565 

58 

.999 

.987 

.957 

.910 

.854 

.797 

.74! 

.685 

.635 

.588 

60 

.999 

.988 

.960 

.915 

.860 

.804 

.749 

.693 

,  644 

.596 

62 

1.000 

.991 

.966 

.928 

.874 

.821 

.767 

.713 

.  664 

.616 

64 

.993 

.971 

.934 

.837 

.836 

.784 

.731 

.683 

.635 

66 

.994 

.976 

.94  3 

.899 

.851 

.802 

.750 

.702 

.655 

63 

.995 

.973 

.947 

,905 

.857 

.309 

.757 

,711 

.664 

70 

.996 

.982 

.954 

.916 

.871 

.825 

.775 

.729 

.683 

72 

.997 

.984 

.957 

.921 

.878 

.832 

.784 

.738 

.693 

74 

.998 

.987 

.965 

.932 

.893 

.851 

.805 

.761 

.716 

76 

.998 

.989 

.969 

.938 

.901 

.861 

.816 

.773 

.728 

78 

.999 

.990 

.971 

.942 

.907 

.868 

.824 

.781 

.737 

80 

.999 

.992 

.974 

.948 

.913 

.376 

.833 

.792 

.749 

32 

.999 

.993 

.978 

.953 

.921 

.386 

.845 

.305 

.762 

84 

.999 

.994 

.979 

.956 

.925 

.390 

.849 

.810 

.768 

86 

1.000 

.995 

.983 

.962 

.934 

.902 

.864 

.827 

.787 

83 

.996 

.985 

.966 

.940 

.908 

.872 

.836 

.796 
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TIES  ALLOWED 


Table  B  7  (Continued) 


M  =  5:  TIES  ALLOWED 


N 


J 

180 
182 
184 
136 
183 
190 
192 
194 
198 
198 
200 
202 
204 
208 
208 
210 
212 
214 
216 
218 
220 
222 
224 
226 
228 
230 
232 
234 
•2  36 
238 
240 

242 

244 

246 

243 
250 
252 
254 
256 
258 
260 
262 
264 
266 
263 


8 

9 

10 

11 

12 

.999 

.998 

.995 

.991 

.984 

.999 

.998 

.996 

.991 

.935 

.999 

.998 

.996 

.992 

.986 

i  .000 

.998 

.996 

.993 

.987 

.999 

.996 

.993 

.998 

.999 

-.997 

.994 

.989 

.999 

.997 

.994 

.989 

.999 

.997 

.994 

.990 

.999 

.998 

.995 

.991 

.999 

.998 

.995 

.  99  i 

.999 

.998 

.995 

.992 

.999 

.998 

.996 

.992 

.999 

.998 

.996 

.992 

.999 

.998 

.996 

.993 

.999 

.998 

.997 

.994 

.999 

999 

.997 

.994 

.999 

999 

.997 

.994 

1 .000 

.999 

.997 

.995 

.999 

.997 

.995 

.999 

.998 

.995 

.999 

.998 

.996 

.999 

.998 

.996 

.999 

.993 

.996 

,999 

.998 

.996 

,999 

.998 

.9/7 

.999 

.993 

.997 

.999 

.999 

.997 

1.000 

.999 

.997 

.999  .997 

.999  -998 

.999  .998 

.999  .993 

.999  .998 

.999  .998 

.999  .998 

.999  .998 

.999  .993 

.999  -999 

.999  .999 

.999  ..999 

1.000  .999 

.999 
..999 
.999 
.999 


K  'll  3-ui.  je*V*VW**fW*l 


Table  B  8 


M  =  6.  N  EVEN:  TIES  ALLOWED 


1  4  6 

0  .000  .000 

2  .007  ..003 

4  .024  .009 

6  .046  019 

3  -083  .036 

10  -125  .056 

12  .158  .072 

14  -214  .102 

16  -273  .135 

13  -308  .156 

20  .362  .190 

22  .419  .227 

34  .461  .257 

26  .512  .294 

28  .557  329 

30  .592  .357 

32  .636  .395 

34  -676  .431 

36  .700  .455 

38  .737  .,93 

40  .771  .529 

42  .788  .548 

44  -813  .573 

46  .839  .610 

48  -854  .631 

50  -872  . 6S7 

52  .888  .681 

34  -901  .701 

56  .916  727 

58  .926  .746 

60  .933  .758 

62  .943  .778 

64  .952  .798 

66  .957  .810 

68  .963  .825 

'0  .969  .84! 

72  -972  .851 

74  -976  .863 

76  .980  .875 

78  -982  .883 

30  .985  .894 

82  .987  .903 

84  .989  .909 

S6  .991  .918 

83  .992  .926 


8 

10 

12 

.000 

.000 

.000 

.001 

.001 

.001 

.005 

.003 

.002 

.010 

.006 

.004 

.019 

.012 

.008 

.031 

.019 

.012 

.040 

.025 

.017 

.058 

.037 

.024 

.078 

.050 

.034 

.091 

.058 

.040 

.112 

.073 

.050 

.13/ 

.090 

.062 

.  157 

.104 

.072 

.  183 

.123 

.086 

.208 

.  140 

.099 

.228 

.155 

.110 

.257 

.177 

.126 

.285 

.198 

.  142 

.304 

.212 

.154 

.333 

.236 

.172 

.364 

.260 

.191 

.380 

.273 

.201 

.407 

.295 

.219 

.436 

.319 

.239 

.457 

.336 

.253 

.481 

.357 

.270 

-505 

.378 

.288 

.526 

.396 

.304 

.552 

.420 

.325 

.573 

.439 

-341 

.587 

.452 

.353 

.610 

.474 

.372 

.632 

.496 

.392 

.647 

.511 

.405 

.666 

.528 

.422 

.687 

.550 

.442 

,700 

.563 

.454 

,717 

.580 

.471 

734 

.598 

.489 

744 

.609 

.500 

761 

.627 

.518 

775 

.643 

.533 

785 

.654 

.545 

799 

.672 

.562 

812 

.687 

.578 
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Table  B  3  (Continued)  H  ■  6,  N  EVES:  TIES  ALLOWED 


N 


J 

9 

6 

8 

10 

12 

90 

.993 

.931 

.820 

.697 

.588 

92 

.999 

.936 

.830 

.709 

.601 

94 

.995 

.943 

.842 

.724 

.617 

96 

.996 

.947 

.850 

.735 

.628 

98 

.997 

.952 

.859 

.746 

.641 

100 

.997 

.956 

.867 

.758 

.653 

102 

.998 

.959 

.374 

.766 

.663 

109 

.998 

.964 

.883 

.779 

.677 

106 

.998 

.967 

.891 

.790 

.690 

108 

.999 

.969 

.896 

.797 

.697 

110 

.999 

.973 

.903 

,808 

.710 

112 

.999 

.975 

.910 

.813 

.722 

119 

.999 

.977 

.914 

.823 

.729 

116 

.999 

.979 

.920 

.832 

-739 

1 18 

1.000 

.982 

.926 

.841 

,751 

120 

.983 

.930 

.348 

.758 

122 

.985 

.934 

.855 

.768 

129 

.986 

.939 

.863 

.777 

126 

.987 

.942 

.868 

.784 

128 

.989 

.946 

.875 

.793 

130 

.990 

.950 

.381 

.301 

132 

.990 

.952 

.885 

.806 

139 

.992 

.956 

.892 

.815 

136 

.993 

.960 

.898 

.823 

138 

,993 

.962 

.902 

.829 

190 

.994 

.965 

.907 

.836 

192 

.994 

.967 

.912 

.843 

199 

.995 

.969 

.916 

.848 

196 

.995 

.971 

.920 

.854 

198 

.996 

.973 

.924 

.860 

150 

.996 

.975 

.928 

.865 

152 

.997 

.977 

.932 

,871 

159 

.997 

.979 

.936 

.876 

156 

.997 

.980 

.938 

.880 

158 

.993 

.981 

.942 

.885 

160 

.998 

.983 

.945 

.89! 

162 

.998 

.984 

.948 

.894 

169 

.998 

.985 

.950 

,899 

166 

.998 

.986 

,954 

.904 

168 

.999 

.987 

,956 

.907 

170 

.999 

.988 

.958 

.911 

172 

.999 

.989 

.960 

.914 

179 

.999 

.990 

.962 

,918 

176 

.999 

.991 

.964 

.922 

173 

.999 

.991 

.966 

.925 

180 

.999 

,992 

.968 

,927 

182 

.999 

.992 

,970 

,931 

34 


Table  B  8  (Continued) 


6.N  EVE;i:  ties  ALLOWED 


M  = 


276 

278 

280 

282 

284 

286 

288 

290 

292 

294 

296 

298 

300 

302 

304 

306 

308 

310 

312 

314 

316 

318 

320 

322 

324 

326 

328 

330 

332 

334 

336 

338 

340 

342 

344 

346 

346 

350 

352 

354 

356 

358 

360 

362 

364 

366 

368 


10 

12 

.998 

.992 

.998 

.992 

.998 

.993 

.996 

.993 

.999 

.993 

.999 

-yy-4 

.999 

.994 

.999 

.994 

.999 

.995 

.999 

.995 

.999 

.995 

.999 

.995 

.  399 

.996 

.999 

.996 

.999 

.996 

.999 

.996 

.999 

.996 

.999 

.997 

.999 

.997 

.999 

.997 

1.000 

,997 

.997 

.997 

,997 

,998 

.998 

.998 

.998 

-.998 

-.998 

.998 

.998 

.998 

.998 

.998 

.999 

.999 

,999 

.999 

.999 

.999 

.999 

.999 

.999 

.999 

.999 

.999 
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Table  B  8  (Continued)  M  «  6,  N  EVEN:  TIES  ALLOWED 


